A Formal Language and Analysis Tool for Black Box Specifications

The black box specification, developed by Harlan Mills, addresses the problem of software errors that result from failing to properly specify a response for an input scenario. Each black box models how an artifact responds to a particular input from its environment. This response depends on both the current input and the entire history of interactions it has had with the environment. We have observed that students find the black box concept useful and comprehensible, but tedious and error-prone as well. In order to enhance the utility and accessibility of this technique, we have developed a formal specification language and analysis tool for black box specifications. The analysis tool verifies whether a black box is a well-formed specification. To this end, it ensures that a response is specified for every possible combination of inputs from the environment, that every condition is logically disjoint with every other condition in the specification, and that every condition in the specification matches at least one potential input scenario. We have evaluated the utility and performance of the tool with two different groups of undergraduate students. INTRODUCTION Many errors in software can be attributed to the failure to properly specify a response for an input scenario. A black box specification [1][3][4] models how a software artifact interacts with its environment. A black box does not represent any internal state of the artifact; it only represents externally observable interactions with its environment. This model can assist a software engineer in ensuring that a response for every input scenario has been specified [3]. We have previously introduced undergraduate students to black box specifications based on the presentation by Prowell et al [3]. While students informed us that the process was useful, they also complained about the tedium involved in constructing and verifying the specifications. To address this problem, we have developed a formal language and modeling tool. The models built in this language can be algorithmically analyzed by the tool to ensure that the black box is a well-formed specification. The analysis tool ensures that a response is specified for every possible combination of inputs from the environment, that every condition is logically disjoint with every other condition, and that every condition matches at least one potential input scenario. The analysis tool is also able to generate an animation of the specified software artifact. This animation is a GUI with a button for each stimulus. It shows both the current response and the entire list of responses issued so far. This allows both the specifier and domain experts to validate the black box against the requirements. We begin by presenting our representation of black box specifications. Next, we describe the verification algorithms employed by the analysis tool. We then discuss the results from our evaluation of the tool with two different groups of undergraduate students. We first analyze the usability and utility of the tool, followed by an analysis of efficiency concerns. We then discuss related work. BLACK BOX SPECIFICATION We call each input from the environment a stimulus and each output a response. A history is a finite-length sequence of alternating stimuli and responses. A well-formed black box specification is a function from the domain of the set of history-stimulus pairs to the range of the set of responses. As a function, it will deterministically respond in the same way for each history-stimulus pair. We represent a black box by a table with three columns: the History Set, the Current Stimulus, and the Response. Each stimulus and response is a discrete symbol. A history set is a (potentially infinite) set of histories. History sets are formally specified using history patterns. Each history pattern maps a history to a true/false value, indicating whether or not that history is a member of the set that the pattern specifies. The formal language for history patterns is constructed as follows. The most primitive language element is the name of a stimulus. This matches any history with exactly one stimulus-response pair that includes the named stimulus. A pair can also be specified in terms of a named response; in this case, the syntax requires the keyword “response” prior to the named response. The keyword “stim” matches any single stimulus-response pair regardless of names. The keyword “any” matches any history, even if it is of zero length. The keyword “none” matches zero-length histories only. These language elements are composed into more general patterns as follows. The “:” operator concatenates any two patterns. The concatenated pattern will match any history that can be divided so that its first part matches the pattern before the “:” while the second part matches the pattern after the “:”. “Count” returns the number of disjoint sub-histories of a history that match its argument. It matches sub-histories greedily; hence, it prefers shorter sub-histories to longer ones in ambiguous situations. The values generated by count can be compared with each other using the standard arithmetic comparison operators (<, >, ≥, ≤, =). They can also be incorporated into linear combinations with addition and subtraction. The construct “includes ” is equivalent to “count ≥ 1”. Since determining membership in a history set is a boolean operation, the standard boolean operators “and”, “or”, and “not” are available. In order to bridge the gap between the pattern language and an application domain, macro patterns can be defined. Each macro is a binding of a name to a pattern. Whenever the name of the macro is used in an expression, the pattern bound to it is used to determine whether a particular history is a match. We will illustrate these language elements using the specification of a simple text editor as an example. The black box specification is given in Table 1. The text editor has three operations: appending a character onto the end of the buffer, removing a character from the end of the buffer, and saving the buffer. To describe these operations we use three stimuli: Char, Back, and Save. Anytime a character is typed, it is appended to the end of the buffer. Hence, the stimulus “Char” can be specified using a single row. Backspacing removes a character only if at least one character is present. If the buffer is empty, backspacing is impossible. Hence, we use two rows to specify the “Back” stimulus. Characters are present only if the number of character additions exceeds the number of successful deletions. Hence, we define the macro “canBackspace” to represent “count Char > count response Del”. Since the stimulus “Back” does not always represent a successful deletion, the history pattern matches the response “Del” in order to ensure that actual deletions are being counted. We specify that the text editor will respond to a request to save the buffer only if the buffer has changed since the last save, which again requires two rows. We will first define the idea of a “change”. We will use that definition to specify “the buffer changed since the last save”. As a change involves either the addition or successful deletion of a character, we define the macro “change” to represent “Char or response Del”. A properly saved buffer cannot have any changes subsequent to being saved. Hence, we define the macro “isSaved” as “any:response Saved:(not includes change)”. We use this macro in both rows to indicate that a save should succeed if and only if the buffer has not been saved. Table 1: Black Box for a Simple Text Editor History Set Stimulus Response